This document is dedicated to conducting the confirmatory performance analyses that were proposed for Experiment 3.

Design and Predictions

Design. The design and analysis is a 2 (difficulty: harder than reference vs. easier than reference) X 2 (difference: moderate vs. extreme) X 2 (transition: repeat vs. switch) within-subjects ANOVA on RTs and error rates.

In retrospect, I’m realizing this design doesn’t make sense because, unlike in Experiment 1, decks aren’t associated with a single level of intensity, but rather the risky deck can always have two outcomes. I think trying to coerce this analysis into the difference X difficulty design would introduce a lot of unnecessary complexity.

I’m taking a step back and thinking about what the actual insights are that I want to get out of this performance data. Are RTs and error rates slower and higher for runs of trials with more switches rather than fewer? Is the switch cost greater when switching occurs less frequently? Thinking about it this way compels me to analyze performance based on numbers of switches in a run, regardless of what deck the run comes from. It might also be interesting to explore how the actual decks influence performance, but that would be more of an exploratory performance analysis.

The analysis I’ll conduct to investigate the above is a regression where number of switches in a run and transition type predict RTs (and errors). The prediction is that RTs will be positively associated with number of switches in a run, RTs will be positively predicted by transition such that switch trials have longer RTs (i.e., the switch cost), and there will be an interaction such that the impact of transition on RT will be greater in runs where switching occurs less frequently.

Results

Response Times

Below is the cleaned data:

d <- read.csv('../../../data/dstClean.csv')
d <- d %>% 
  filter(transition != 'startBlock')
N <- d %>% 
  group_by(subject) %>% 
  summarize(n()) %>% 
  nrow(.)
d

The sample size is 131.

Visualize the Results

subjectCellMeans <- d %>% 
  filter(transition != 'startBlock') %>% 
  group_by(subject, outcomeSwitch, transition) %>% 
  summarize(rtime = mean(cuedRt)) 
## `summarise()` has grouped output by 'subject', 'outcomeSwitch'. You can
## override using the `.groups` argument.
subjectCellMeans %>% 
  group_by(outcomeSwitch, transition) %>% 
  summarize(rt = mean(rtime), se = sd(rtime) / sqrt(N)) %>% 
  ggplot(aes(x = factor(outcomeSwitch), y = rt, group = transition)) +
  geom_point(size = 2) +
  geom_line(aes(linetype = transition)) +
  geom_errorbar(aes(ymin = rt - se, ymax = rt + se), width = 0.5) +
  labs(
    x = 'Number of Switches in a Run',
    y = 'Response Time (ms)'
  ) + 
  scale_linetype_discrete(name = 'Transition Type', labels = c('Repeat', 'Switch')) +
  theme_bw() +
  theme(legend.position = c(.9,.5),
        panel.grid = element_blank())
## `summarise()` has grouped output by 'outcomeSwitch'. You can override using the
## `.groups` argument.

Fit a linear model

d$outcomeSwitchC <- d$outcomeSwitch - 8
d$transitionE <- ifelse(d$transition == 'repeat', -0.5, 0.5)
m1 <- lm(cuedRt ~ transitionE * outcomeSwitchC, data = d)
plot(m1)

summary(m1)
## 
## Call:
## lm(formula = cuedRt ~ transitionE * outcomeSwitchC, data = d)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -1134.0  -276.0  -111.4   140.0  3648.0 
## 
## Coefficients:
##                            Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                999.2967     1.7585 568.257  < 2e-16 ***
## transitionE                258.1030     3.5171  73.386  < 2e-16 ***
## outcomeSwitchC               5.8015     0.5737  10.112  < 2e-16 ***
## transitionE:outcomeSwitchC  -3.1417     1.1475  -2.738  0.00618 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 447.4 on 75874 degrees of freedom
## Multiple R-squared:  0.0857, Adjusted R-squared:  0.08567 
## F-statistic:  2371 on 3 and 75874 DF,  p-value: < 2.2e-16
confint(m1)
##                                 2.5 %       97.5 %
## (Intercept)                995.850019 1002.7434357
## transitionE                251.209550  264.9963834
## outcomeSwitchC               4.676962    6.9260493
## transitionE:outcomeSwitchC  -5.390813   -0.8926388
newdata <- data.frame(outcomeSwitch = rep(unique(d$outcomeSwitch), 2), transition = c(rep(-0.5, length(unique(d$outcomeSwitch))), rep(0.5, length(unique(d$outcomeSwitch)))))
newdata$proba <- m1$coefficients[1] + newdata$outcomeSwitch * m1$coefficients[3] + newdata$transition * m1$coefficients[2] + newdata$transition * newdata$outcomeSwitch * m1$coefficients[4]
newdata$transitionE <- newdata$transition
newdata$outcomeSwitchC <- newdata$outcomeSwitch - 8
newdata$transition <- ifelse(newdata$transition == -0.5, 'repeat', 'switch')
newdata <- cbind(newdata, predict(m1, newdata, interval = 'predict'))
ggplot(newdata, aes(x = factor(outcomeSwitch), y = fit, group = transition)) + 
  geom_jitter(data = subjectCellMeans, aes(x = factor(outcomeSwitch), y = rtime, color = factor(transition)), alpha = .2, width = .2, height = 0, show.legend = FALSE) +
  geom_line(size = 2, aes(color = factor(transition))) +
  #geom_ribbon(aes(ymin = lwr, ymax = upr, fill = factor(transition)), alpha = .1, show.legend = FALSE) +
  scale_color_manual(name = 'Transition Type', values = c(`repeat` = 'blue', `switch` = 'red'), labels = c('Repeat', 'Switch')) +
  #scale_fill_manual(values = c(`repeat` = 'blue', `switch` = 'red')) +
  labs(
    x = 'Number of Switches in a Run',
    y = 'Response Time (ms)',
    caption = 'Lines reflect predictions from regression. Points reflect subject-wise cell means.'
  ) +
  theme_bw() +
  theme(legend.position = c(.8,.85),
        panel.grid = element_blank())

Centering at high number of switches per run (12)

d$outcomeSwitchC <- d$outcomeSwitch - 12
d$transitionE <- ifelse(d$transition == 'repeat', -0.5, 0.5)
m1 <- lm(cuedRt ~ transitionE * outcomeSwitchC, data = d)
summary(m1)
## 
## Call:
## lm(formula = cuedRt ~ transitionE * outcomeSwitchC, data = d)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -1134.0  -276.0  -111.4   140.0  3648.0 
## 
## Coefficients:
##                             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                1022.5027     2.8463 359.243  < 2e-16 ***
## transitionE                 245.5361     5.6925  43.133  < 2e-16 ***
## outcomeSwitchC                5.8015     0.5737  10.112  < 2e-16 ***
## transitionE:outcomeSwitchC   -3.1417     1.1475  -2.738  0.00618 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 447.4 on 75874 degrees of freedom
## Multiple R-squared:  0.0857, Adjusted R-squared:  0.08567 
## F-statistic:  2371 on 3 and 75874 DF,  p-value: < 2.2e-16
confint(m1)
##                                  2.5 %       97.5 %
## (Intercept)                1016.924068 1028.0814314
## transitionE                 234.378699  256.6934251
## outcomeSwitchC                4.676962    6.9260493
## transitionE:outcomeSwitchC   -5.390813   -0.8926388

And centering at low number of switches per run (4)

d$outcomeSwitchC <- d$outcomeSwitch - 4
d$transitionE <- ifelse(d$transition == 'repeat', -0.5, 0.5)
m1 <- lm(cuedRt ~ transitionE * outcomeSwitchC, data = d)
summary(m1)
## 
## Call:
## lm(formula = cuedRt ~ transitionE * outcomeSwitchC, data = d)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -1134.0  -276.0  -111.4   140.0  3648.0 
## 
## Coefficients:
##                            Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                976.0907     2.9356 332.505  < 2e-16 ***
## transitionE                270.6699     5.8711  46.102  < 2e-16 ***
## outcomeSwitchC               5.8015     0.5737  10.112  < 2e-16 ***
## transitionE:outcomeSwitchC  -3.1417     1.1475  -2.738  0.00618 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 447.4 on 75874 degrees of freedom
## Multiple R-squared:  0.0857, Adjusted R-squared:  0.08567 
## F-statistic:  2371 on 3 and 75874 DF,  p-value: < 2.2e-16
confint(m1)
##                                 2.5 %      97.5 %
## (Intercept)                970.337005 981.8444044
## transitionE                259.162471 282.1772706
## outcomeSwitchC               4.676962   6.9260493
## transitionE:outcomeSwitchC  -5.390813  -0.8926388

Looking at the relationship between switches in a run and mean RT (dropping transition)

m2 <- lm(cuedRt ~ outcomeSwitch, data = d)
slope <- round(m2$coefficients[2], 2)

summary(m2)
## 
## Call:
## lm(formula = cuedRt ~ outcomeSwitch, data = d)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -1054.5  -302.0  -110.8   162.4  3580.5 
## 
## Coefficients:
##               Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   820.1033     4.6697  175.62   <2e-16 ***
## outcomeSwitch  21.9433     0.5465   40.15   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 463 on 75876 degrees of freedom
## Multiple R-squared:  0.0208, Adjusted R-squared:  0.02079 
## F-statistic:  1612 on 1 and 75876 DF,  p-value: < 2.2e-16
confint(m2)
##                   2.5 %    97.5 %
## (Intercept)   810.95073 829.25588
## outcomeSwitch  20.87207  23.01448
sMeans <- d %>% 
  group_by(subject, outcomeSwitch) %>% 
  summarize(rtime = mean(cuedRt)) 
## `summarise()` has grouped output by 'subject'. You can override using the
## `.groups` argument.
sMeans %>% 
  group_by(outcomeSwitch) %>% 
  summarize(rt = mean(rtime), se = sd(rtime) / sqrt(N)) %>% 
  ggplot(aes(x = factor(outcomeSwitch), y = rt, group = 1)) +
  geom_line() +
  geom_point(size = 2.5) +
  geom_jitter(data = sMeans, aes(x = factor(outcomeSwitch), y = rtime), alpha = .2, width = .1, height = 0) +
  geom_errorbar(aes(ymin = rt - se, ymax = rt + se), width = 0.5) +
  annotate('text', x = 4, y = 1500, label = paste('b = ', slope, sep = '')) +
  labs(
    x = 'Number of Switches in a Run',
    y = 'Response Time (ms)'
  ) +
  theme_bw() +
  theme(panel.grid = element_blank())

Error Rates

Below is the cleaned data:

d <- read.csv('../../../data/dstCleanErrors.csv')

d <- d %>% 
  filter(d$transition != 'startBlock')

N <- d %>% 
  group_by(subject) %>% 
  summarize(n()) %>% 
  nrow(.)
d

The sample size is 131.

Visualize the Results

subjectCellMeans <- d %>% 
  group_by(subject, outcomeSwitch, transition) %>% 
  summarize(error = mean(error)) 
## `summarise()` has grouped output by 'subject', 'outcomeSwitch'. You can
## override using the `.groups` argument.
subjectCellMeans %>% 
  group_by(outcomeSwitch, transition) %>% 
  summarize(err = mean(error), se = sd(error) / sqrt(N)) %>% 
  ggplot(aes(x = factor(outcomeSwitch), y = err, group = transition)) +
  geom_point(size = 2) +
  geom_line(aes(linetype = transition)) +
  geom_errorbar(aes(ymin = err - se, ymax = err + se), width = 0.5) +
  labs(
    x = 'Number of Switches in a Run',
    y = 'Error Rate'
  ) + 
  scale_linetype_discrete(name = 'Transition Type', labels = c('Repeat', 'Switch')) +
  theme_bw() +
  theme(legend.position = c(.2,.8),
        panel.grid = element_blank())
## `summarise()` has grouped output by 'outcomeSwitch'. You can override using the
## `.groups` argument.

Fit a linear model

d$outcomeSwitchC <- d$outcomeSwitch - 8
d$transitionE <- ifelse(d$transition == 'repeat', -0.5, 0.5)
m1 <- lm(error ~ transitionE * outcomeSwitchC, data = d)
plot(m1)

summary(m1)
## 
## Call:
## lm(formula = error ~ transitionE * outcomeSwitchC, data = d)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.05023 -0.04568 -0.02801 -0.02644  0.97482 
## 
## Coefficients:
##                             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)                0.0352388  0.0006946  50.730  < 2e-16 ***
## transitionE                0.0163380  0.0013893  11.760  < 2e-16 ***
## outcomeSwitchC             0.0007254  0.0002266   3.202  0.00137 ** 
## transitionE:outcomeSwitchC 0.0008219  0.0004531   1.814  0.06972 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.1855 on 83695 degrees of freedom
## Multiple R-squared:  0.002519,   Adjusted R-squared:  0.002483 
## F-statistic: 70.44 on 3 and 83695 DF,  p-value: < 2.2e-16
confint(m1)
##                                    2.5 %      97.5 %
## (Intercept)                 3.387733e-02 0.036600305
## transitionE                 1.361500e-02 0.019060940
## outcomeSwitchC              2.813563e-04 0.001169499
## transitionE:outcomeSwitchC -6.626814e-05 0.001710018
newdata <- data.frame(outcomeSwitch = rep(unique(d$outcomeSwitch), 2), transition = c(rep(-0.5, length(unique(d$outcomeSwitch))), rep(0.5, length(unique(d$outcomeSwitch)))))
newdata$proba <- m1$coefficients[1] + newdata$outcomeSwitch * m1$coefficients[3] + newdata$transition * m1$coefficients[2] + newdata$transition * newdata$outcomeSwitch * m1$coefficients[4]
newdata$transitionE <- newdata$transition
newdata$outcomeSwitchC <- newdata$outcomeSwitch - 8
newdata$transition <- ifelse(newdata$transition == -0.5, 'repeat', 'switch')
newdata <- cbind(newdata, predict(m1, newdata, interval = 'predict'))
ggplot(newdata, aes(x = factor(outcomeSwitch), y = fit, group = transition)) + 
  geom_jitter(data = subjectCellMeans, aes(x = factor(outcomeSwitch), y = error, color = factor(transition)), alpha = .2, width = .2, height = 0, show.legend = FALSE) +
  geom_line(size = 2, aes(color = factor(transition))) +
  #geom_ribbon(aes(ymin = lwr, ymax = upr, fill = factor(transition)), alpha = .1, show.legend = FALSE) +
  scale_color_manual(name = 'Transition Type', values = c(`repeat` = 'blue', `switch` = 'red'), labels = c('Repeat', 'Switch')) +
  #scale_fill_manual(values = c(`repeat` = 'blue', `switch` = 'red')) +
  labs(
    x = 'Number of Switches in a Run',
    y = 'Error Rate',
    caption = 'Lines reflect predictions from regression. Points reflect subject-wise cell means.'
  ) +
  ylim(0,.2) +
  theme_bw() +
  theme(legend.position = c(.8,.85),
        panel.grid = element_blank()) 
## Warning: Removed 40 rows containing missing values (geom_point).

Looking at the relationship between switches in a run and mean RT (dropping transition)

m2 <- lm(error ~ outcomeSwitch, data = d)
slope <- round(m2$coefficients[2], 5)

summary(m2)
## 
## Call:
## lm(formula = error ~ outcomeSwitch, data = d)
## 
## Residuals:
##      Min       1Q   Median       3Q      Max 
## -0.04605 -0.03916 -0.03572 -0.03055  0.97462 
## 
## Coefficients:
##                Estimate Std. Error t value Pr(>|t|)    
## (Intercept)   0.0219398  0.0017943  12.228   <2e-16 ***
## outcomeSwitch 0.0017225  0.0002089   8.247   <2e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.1856 on 83697 degrees of freedom
## Multiple R-squared:  0.000812,   Adjusted R-squared:  0.0008 
## F-statistic: 68.02 on 1 and 83697 DF,  p-value: < 2.2e-16
confint(m2)
##                     2.5 %      97.5 %
## (Intercept)   0.018423087 0.025456604
## outcomeSwitch 0.001313115 0.002131833
sMeans <- d %>% 
  group_by(subject, outcomeSwitch) %>% 
  summarize(error = mean(error)) 
## `summarise()` has grouped output by 'subject'. You can override using the
## `.groups` argument.
sMeans %>% 
  group_by(outcomeSwitch) %>% 
  summarize(err = mean(error), se = sd(error) / sqrt(N)) %>% 
  ggplot(aes(x = factor(outcomeSwitch), y = err, group = 1)) +
  geom_line() +
  geom_point(size = 2.5) +
  geom_jitter(data = sMeans, aes(x = factor(outcomeSwitch), y = error), alpha = .2, width = .1, height = 0) +
  geom_errorbar(aes(ymin = err - se, ymax = err + se), width = 0.5) +
  annotate('text', x = 4, y = .3, label = paste('b = ', slope, sep = '')) +
  labs(
    x = 'Number of Switches in a Run',
    y = 'Error Rate'
  ) +
  theme_bw() +
  theme(panel.grid = element_blank())

 

Analysis Homepage

A work by Dave Braun

dab414@lehigh.edu